Method and apparatus of creating a perceptual harmony map

ABSTRACT

The present invention generally relates to a method and apparatus for creating an harmony map of an image. It also relates to a method and apparatus for assessing the quality of an image. The invention also relates to a method and apparatus for assessing the quality of an image such that an assessment score of the image is obtained from a perceptual map of the image created from the previous method. As an interesting tool for content creator and in the goal of maximizing the artistic effect, the proposed method provides perceptual harmony-guided quality map as well as a score of disharmony of a picture without reference image. The method links two distinct subjects: perceptual quality metric and harmonious hue templates in order to provide new tools to content creator whatever the expertise level.

This application claims the benefit, under 35 U.S.C. §365 ofInternational Application PCT/EP2014/056069, filed Mar. 26, 2014, whichwas published in accordance with PCT Article 21(2) on Oct. 2, 2014 inEnglish and which claims the benefit of European patent application No.13305394.2, filed Mar. 28, 2013.

1. FIELD OF INVENTION

The present invention generally relates to a method and apparatus forcreating an harmony map of an image. It also relates to a method andapparatus for assessing the quality of an image.

2. TECHNICAL BACKGROUND

When manipulating, editing, improving image, the best quality as well asa certain artistic intent are usually the finality. Nevertheless,although the issues related to objective quality assessment have beenwidely studied in the context of low level artefacts (such as blur,blockiness, jitter, . . . ), the artistic intent is a problem somehowmore tricky and subjective leading to high difficulties in modeling orgeneralization.

As an intermediate indicator, aesthetic quality metric based on features(colorfulness, line orientation, shape, . . . ) intuitively related tobeauty and rules (composition, rules-of-third, skyline, . . . ) areshowing up recently in the community.

Depending on the application context, some approaches take advantage ofa reference source or do the best effort without any reference whenproviding an absolute quality measurement.

The problem solves by the invention is to define a method for assessingthe color harmony of an image and for assessing what the quality of animage (picture) is.

3. SUMMARY OF THE INVENTION

The invention solves this problem by defining a method for creating aperceptual harmony map of an image which comprises:

-   -   calculating a multi-resolution decomposition of the image,    -   at each resolution level,    -   for each harmonious color template, computing an optimal        rotation angle and an harmony distance for the hue value        associated with each pixel of the image at the resolution level,        and computing a harmony distance map by weighting the sum of        said harmony distances,    -   calculating a contrast map and an entropy activity map,    -   obtaining a perceptual harmony map by integrating together the        contrast map the entropy activity map and the harmony distance        map, and    -   obtaining the perceptual map by accumulating the perceptual        harmony maps.

The invention also relates to a method for assessing the quality of animage characterized in that an assessment score of the image is obtainedfrom a perceptual map of the image created from the previous method.

As an interesting tool for content creator and in the goal of maximizingthe artistic effect, the proposed method provides perceptualharmony-guided quality map as well as a score of disharmony of a picturewithout reference image.

The method links two distinct subjects: perceptual quality metric andharmonious hue templates in order to provide new tools to contentcreator whatever the expertise level.

The multi-resolution decomposition of the image is processed tocalculate a perceptual harmony map at different resolutions of the imagein order to mimic the human visual system.

Moreover, each perceptual harmony map is obtained by applying maskingfunctions to the harmony distance map or, in other words, by integratingtogether the contrast map, the entropy activity map and the harmonydistance map. Such masking of the harmony distance map providesinformation regarding local and global visibility.

The specific nature of the invention as well as other objects,advantages, features and uses of the invention will become evident fromthe following description of a preferred embodiment taken in conjunctionwith the accompanying drawings.

4. LIST OF FIGURES

The embodiments will be described with reference to the followingfigures:

FIG. 1 represents color templates;

FIG. 2 depicts a flowchart of the processing method for creating aperceptual harmony map of an image; and

FIG. 3 depicts a device according comprising means to implement themethod for creating a perceptual harmony map of the image.

5. DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

Harmony is a subjective concept whose difficulties lie on its definitionand its measurement. Fedorovskaya et al. (E. Fedorovskaya, C.Neustaedter, and W. Hao. Image harmony for consumer images. In ImageProcessing, 2008. ICIP 2008. 15^(th) IEEE International Conference on,pages 121-124, October) investigated through a series of experiments theidentification of some image features involving harmony (dis)comfort.They found out that edge contrast, average lightness, range of lightness. . . may influence global harmony appreciation. Despite this previousreference, most of harmony investigations in image processing fieldrelates to color combinations and complementaries. Starting with Ittencontrast measurement where harmonious doublets and triplets of color aredefined (J. Itten. The art of color: the subjective experience andobjective rationale of color. Van Nostrand Reinhold, New York, 1973.),Matsuda (Y. Matsuda. Color design, Asakura Shoten, 1995) extended thiswork by defining HSV (Hue, Saturation, Value)-based templates thatpredict set of harmonious hues within an image.

An harmonious color template is thus a set of HSV values (hue,saturation and value) that are considered as rendering/reflecting aglobal harmonious effect when present at the same time. Heightharmonious color templates T_(m) (mε{i, I, L, T, V, X, Y, J}) are usedas depicted on FIG. 1. The harmonious color templates O and N are notused. Each harmonious color template T_(m) is made of differentportions/sectors to handle color complementary, color orthogonality andset of close hues. Each sector has a center α_(m) and a size w_(m). Aharmonious color template may be turned around the hue wheel. Thus anharmonious color template is defined by a template type and a sectorangle α_(m).

A complete implementation of the invention is depicted in FIG. 2. Onecan notice that the following method can be extended to video source byapplying same process to consecutive frames.

At step 10, a multi-resolution decomposition of the image is calculated.

For example, a Discrete Wavelet Transform (DWT) which has proven to beefficient both in term of prediction and computation performances (A.Ninassi, O. LeMeur, P. Le Callet, and D. Barba. On the performance ofhuman visual system based image qual-ity assessment metric using waveletdomain. Pages 680610-680610-12, 2008.). A CDF 9/7(Cohen-Daubechies-Feauveau) kernel may be used.

Each decomposition level comprises three orientation subbands(horizontal, vertical, and oblique frequencies). The spatial frequencyrange of a decomposition level lε[0; L−1] is [2^(−(l+1))·f_(max);2⁻¹·f_(max)] where f_(max) is the maximum spatial frequency of an inputimage Im.

At step 20, at each resolution level l, for each harmonious colortemplate T_(m), an optimal rotation angle α_(m,l) is computed and anharmony distance HD_(m,l)(h) is computed for the hue value h associatedwith each pixel (x,y) of the image Im at the resolution level l. Aharmony distance map HDMap_(l)(x,y) is then calculated for each pixel ofthe image Im at the resolution level l by weighting the sum of saidharmony distances HD_(m,l)(h).

According to an embodiment, the optimal rotation angle α_(m,l) which iscomputed for a harmonious color template T_(m) minimizes the energyE_(α) _(m,l) defined by the Kullback-Liebler divergence between theoriginal hue distribution M_(h,l) of the image Im at the resolutionlevel l and the distribution P_(h,l)(m, α_(m,l)) of the harmonious colortemplate T_(m) for the angle α_(m,l) such as described, for example byBaveye et al. (Y. Baveye, F. Urban, C. Chamaret, V. Demoulin, and P.Hellier. Saliency-guided consistent color harmonization. In S. Tominaga,R. Schettini, and A. Trmeau, editors, Computational Color Imaging,volume 7786 of Lecture Notes in Computer Science, pages 105-118.Springer Berlin Heidelberg, 2013).

Note that the distributions M_(h,l) and P_(h,l)(m, l, α_(m,l)) may beconsidered as being histograms with 360 bins.

The distributions M_(h,l), i.e. the color histogram M_(h,l), is computedin the HSV space as a normalized hue distribution weighted by saturationS(x,y) and value V(x,y):

$M_{h,l} = {\frac{1}{\Sigma_{({x,y})}{S\left\lbrack {x,y} \right\rbrack}*{V\left\lbrack {x,y} \right\rbrack}}*{\sum\limits_{{({x,y})} \in {\{{{{({u,v})}\backslash{H{\lbrack{u,v}\rbrack}}} = h}\}}}\;{{S\left\lbrack {x,y} \right\rbrack}*{V\left\lbrack {x,y} \right\rbrack}}}}$

where (x,y) is a pixel belonging to the image Im at each resolutionlevel l.

Mathematically speaking, the energy E_(α) _(m,l) is given by:

$E_{\alpha_{m,l}} = {\min\limits_{\alpha}{\sum\limits_{h}\;{M_{h,l}x\;{\ln\left( \frac{M_{h,l}}{P_{h,l}\left( {m,\alpha} \right)} \right)}}}}$

with the distribution P_(h,l)(m, α) for a harmonious color templateT_(m) with n sectors is given by:

${P_{h,l}\left( {m,\alpha} \right)} = {\sum\limits_{k = 1}^{n}\;{S_{k}\left( {m,\alpha_{k}} \right)}}$

and a distribution of a sector S_(k)(m, α_(k)) given for each bin valuei by:

$\quad\left\{ \begin{matrix}{\exp\left( {- \frac{1}{1 - \left( \frac{2\; x{{i - \alpha_{k}}}}{w_{k}} \right)}} \right)} & {{sih} \in \left\lbrack {{\alpha_{k} - \frac{w_{k}}{2}},{\alpha_{k} + \frac{w_{k}}{2}}} \right\rbrack} \\0 & {otherwise}\end{matrix} \right.$

The invention is not limited by the way the distribution is defined.

According to an embodiment, the harmony distance HD_(m,l)(h) is anarc-length distance on the hue wheel (measured in degrees) given by:

${{HD}_{m,l}(h)} = \left\{ \begin{matrix}0 & {{{if}\mspace{14mu}{{h - \alpha_{k}}}} < \frac{w_{k}}{2}} \\{{{h - \alpha_{k}}} - \frac{w_{k}}{2}} & {otherwise}\end{matrix} \right.$

where ∥•∥ is the arc-length distance, α_(k) is the closest sectorcenter, w_(k) its size in degrees and h is the hue value of a pixel ofthe image Im at a resolution level l.

According to an embodiment, the harmony distance map HDMap_(l)(x,y) isthe sum of said harmony distances HD_(m,l)(h) weighted by the relativeenergy calculated over the harmonious color templates.

Mathematically speaking, the harmony distance map HDMap_(l)(x,y) isgiven by:

${{HDMap}_{l}\left( {x,y} \right)} = {\sum\limits_{T_{m}}\;{\frac{E_{Total} - E_{\alpha_{m,l}}}{E_{Total}}{{xHD}_{m,l}(h)}}}$

At step 30, a contrast map (M_(l,o)(m,n) is calculated for eachresolution level l.

The contrast map models the visibility change of the signal due tocontrast values.

According to an embodiment, when a DWT is used at step 10, the contrastmasking is defined as the wavelet transformed values at site (m,n),weighted by the CSF function that describes, at a resolution level l thevariations in visual sensitivity to the spatial frequency andorientation o.

Mathematically speaking, given a CSF value N_(l,o) as the mean value ofa 2D CSF from Daly (S. Daly. Digital images and human vision. ChapterThe visible differences predictor: an algorithm for the assessment ofmage) over the spatial frequencies covered by a subband S_(l,o), thecontrast masking map is given by:CM_(l,o)(m,n)=s _(l,o)(m,n)·N _(l,o)

At step 40, an entropy activity map SML_(l)(x,y) is calculated for eachresolution level l.

The entropy activity map reflects the uncertainty of the image such asin texture area.

According to an embodiment, the entropy activity map SML_(l) isevaluated by the computation of entropy on a n-by-n neighborhood.

Preferably, the entropy activity map SML_(l) is evaluated by computingas the minimum of the directional gradients (horizontal, vertical,diagonal) for each pixel (x,y) of the image Im. Such a semi-localactivity map SML_(l) is then given by

${{SML}_{l}\left( {x,y} \right)} = {\min\left( {\frac{\partial I}{\partial x},\frac{\partial I}{\partial y},{\frac{1}{2} \cdot \left( {\frac{\partial I}{\partial x} + \frac{\partial I}{\partial y}} \right)}} \right)}$

where I is the input image.

The harmony distance maps are computed at differents resolutions levelto match the HVS model.

At step 50, for each resolution level l, the contrast map CM_(l,o)(m,n), the entropy activity map SML_(l)(x,y) and the harmony distancemap HDMap_(l)(x,y) are integrated together to get a perceptual harmonymap PHDMap_(l)(x,y).

According to an example, said perceptual harmony distance mapPHDMap_(l)(x,y) is given by:

${{PHDMap}_{l}\left( {m,n} \right)} = \frac{{HDMap}_{l}\left( {m,n} \right)}{1 + {\frac{1}{2}\left( {{\Sigma_{o}{{CM}_{l,o}\left( {m,n} \right)}} + {{SML}_{i}\left( {m,n} \right)}} \right)}}$

where (m,n) are the coordinates corresponding to the pixel (x,y) of theimage Im at resolution level l.

At step 60, the perceptual harmony maps PHDMap_(l)(m,n) are accumulatedto built a perceptual map PHMap (x,y).

Mathematically speaking, the perceptual map PHMap (x,y) is given by:

${{PHMap}\left( {x,y} \right)} = {\sum\limits_{l}\;{{PHDMap}_{l}\left( {m,n} \right)}}$

Note the perceptual map PHMap (x,y) is visually close to a harmonydistance map computed by accumulating the harmony distance mapHDMap_(l)(x,y), but integrates masking effects by decreasing impact ofsmall group of pixels.

According to an aspect of the invention, a method for assessing thequality of an image is defined by obtaining a rating or assessment scoreR of the image from the perceptual map PHMap (x,y) (step 70 in FIG. 2).

According to an embodiment, the rating R is computed from by means of aMinkowski summation given by:

$R = \left( {\frac{1}{W \cdot H}{\sum\limits_{y}^{H}\;{\sum\limits_{x}^{W}\;\left( {{PHMap}\left( {x,y} \right)} \right)^{\beta}}}} \right)^{\frac{1}{\beta}}$

where W and H are respectively the width and height of the image Im andβ is a parameter set for example to 2.

The color information of an image at a resolution level is carried bythe lowest subband.

Consequently, according to a variant of the method, the contrast andentropy activity map are integrated with the harmony distance map at agiven resolution level only for the lowest subband of this resolutionlevel.

FIG. 3 represents an exemplary architecture of a processing device 300according to a specific and non limiting embodiment. The processingdevice can be for example a tablet, a PDA or a cell phone. Processingdevice 300 comprises following elements that are linked together by adata and address bus 30:

-   -   a microprocessor 31 (or CPU), which is, for example, a DSP (or        Digital Signal Processor);    -   a ROM (or Read Only Memory) 32;    -   a RAM (or Random Access Memory) 33;    -   one or several Input/Output interface(s) 35, for example a        keyboard, a mouse; and    -   a battery 36.

Each of these elements of FIG. 3 are well known by those skilled in theart and won't be disclosed further. The processing device 300 maycomprise display means such as a screen for displaying the processedimages. Algorithm of the processing method according to the inventionare stored in the ROM 32. RAM 33 comprises in a register, the programexecuted by the CPU 31 and uploaded after switch on of the processingdevice 300. When switched on, the CPU 21 uploads the program in the RAMand executes the corresponding instructions. The images to be processedare received on one of the Input/Output interfaces 35. One of theInput/Output interface 35 is adapted to transmit the images processedaccording to the invention.

According to variants, processing devices 300 compatible with theinvention are implemented according to a purely hardware realisation,for example in the form of a dedicated component (for example in an ASIC(Application Specific Integrated Circuit) or FPGA (Field-ProgrammableGate Array) or VLSI (Very Large Scale Integration) or of severalelectronic components integrated into a device or even in a form of amix of hardware elements and software elements.

While not explicitly described, the present embodiments and variants maybe employed in any combination or sub-combination.

The invention claimed is:
 1. A method performed by a device for creatinga perceptual harmony map of an image from at least one harmonious colortemplate defined by a center and at least a sector wherein the methodcomprises: calculating, by a processor, a multi-resolution decompositionof the image, at each resolution level, for each harmonious colortemplate, computing, by said processor, an optimal rotation angle ofsaid harmonious color template around its center and an harmony distancefor the hue value associated with each pixel of the image at theresolution level, said harmony distance quantifying the distance betweensaid hue value and a sector of said harmonious color template, andcomputing a harmony distance map by weighting the sum of harmonydistances calculated for the harmonious color templates calculating, bysaid processor, a contrast map and an entropy activity map reflectingthe uncertainty of the image, obtaining, by said processor, a perceptualharmony map by integrating together the contrast map the entropyactivity map and the harmony distance map, and obtaining, by saidprocessor, the perceptual map by accumulating the perceptual harmonymaps calculated over the different resolution levels.
 2. The method ofclaim 1, wherein an assessment score of the image is obtained, by saidprocessor, from the perceptual map.
 3. A processor readablenon-transitory medium having stored therein instructions for causing aprocessor to perform at least the steps of the method according toclaim
 1. 4. A processor readable non-transitory medium having storedtherein instructions for causing a processor to perform at least thesteps of the method according to claim
 2. 5. Non-transitory storagemedium carrying instructions of program code for executing steps of themethod according to claim 1, when said program is executed on acomputing device.
 6. Device for creating a perceptual harmony map of animage from at least one harmonious color template defined by a centerand at least a sector wherein it comprises a processor configured to:calculate a multi-resolution decomposition of the image, at eachresolution level, for each harmonious color template, compute an optimalrotation angle of said harmonious color template around its center andan harmony distance for the hue value associated with each pixel of theimage at the resolution level, said harmony distance quantifying thedistance between said hue value and a sector of said harmonious colortemplate, and compute a harmony distance map by weighting the sum ofharmony distances calculated for the harmonious color templatescalculate a contrast map and an entropy activity map reflecting theuncertainty of the image, obtain a perceptual harmony map by integratingtogether the contrast map the entropy activity map and the harmonydistance map, and obtain the perceptual map by accumulating theperceptual harmony maps calculated over the different resolution levels.7. A device for assessing the quality of an image, wherein it comprisesa processor configured to obtain an assessment score of the image from aperceptual map of the image, the device wherein the processor is furtherconfigured to: calculate a multi-resolution decomposition of the image,at each resolution level, for each harmonious color template, compute anoptimal rotation angle of said harmonious color template around itscenter and an harmony distance for the hue value associated with eachpixel of the image at the resolution level, said harmony distancequantifying the distance between said hue value and a sector of saidharmonious color template, and compute a harmony distance map byweighting the sum of harmony distances calculated for the harmoniouscolor templates calculate a contrast map and an entropy activity mapreflecting the uncertainty of the image, obtain a perceptual harmony mapby integrating together the contrast map the entropy activity map andthe harmony distance map, and obtain the perceptual map by accumulatingthe perceptual harmony maps calculated over the different resolutionlevels.